Red Hat Enterprise Linux 7 Troubleshooting

Being Proactive, Part 2

Module Topics

  • Managing Configuration Files

  • Red Hat Network Satellite Configuration Channels

  • Change Management Best Practices

  • Configuration Management

Managing Configuration Files

  • When editing configuration files to troubleshoot problems, back up the files before you touch them.

    • Use a revision control system

    • Or, make a copy by hand and include the current date and time in the file name.

      Example
      [root@server1 etc]# cp bofh.conf bofh.conf.$(date +%Y%m%d-%H%M)
  • Test after each and every single change.

    • If the change was successful, congratulations!

    • If not successful, revert to the previous version and try something else.

    • Repeat this process until the problem is solved.

Managing Configuration Files

Using diff
  • Use diff to compare the contents of two files.

  • Example: Comparing station1 (malfunctioning) and station20 (functioning):

    [student@server1 ~]$ diff file.conf-station1 file.conf-station20
    1c1 (1)
    < Hostname = station1
    ---
    > Hostname = station20
    2c2 (2)
    < Setting1 = a
    ---
    > Setting1 = A
    3a4 (3)
    > Setting2 = B
    5d5 (4)
    < Setting4 = D
  • diff shows four places where the configurations differ:

1The Hostname variable is different in each file.
2The Setting1 variable is set to a on station1 and A on station20.
3Setting2 is set on station20, but not on station1.
4Setting4 is set on station1, but not on station20.

Managing Configuration Files

Using Unified Diff Format
  • Unified diff format helps the patch utility more accurately apply changes.

  • If you have several similar files that need the same changes, you can do the following:

    1. Make the change once.

    2. Store a unified diff comparing the changed file to an unchanged file.

    3. Use patch to apply your changes to multiple files.

      [student@server1 ~]$ diff -u file.conf-station1 file.conf-station20
      1c1
      --- file.conf-station1 2007-01-03 18:36:36.000000000 -0800
      +++ file.conf-station20 2007-01-02 21:10:30.000000000 -0800
      @@ -1,4 +1,4 @@
      -Hostname = station1
      -Setting1 = a
      +Hostname = station20
      +Setting1 = A
      +Setting2 = B
       Setting3 = C
      -Setting4 = D

Managing Configuration Files

Using patch
  1. Store the output of a diff -u in a file.

  2. Run a command like the following:

    Example: Make file.conf-station1 look like file.conf-station20
    [student@server1 ~]$ patch -b file.conf-station1 file.conf.patch
    • It would be advisable to first edit file.conf.patch and remove the two lines describing the Hostname variable.

      • Those should remain different between systems.

    • If anything terrible happens, patch -b automatically creates a backup of each file it changes.

    • Backups are given the .orig extension.

Reference
  • diff(1) and patch(1) man pages

Managing Configuration Files

Using Git
  • Revision control systems (or RCS) are sets of tools that allow administrators to easily maintain different versions of files.

  • Historically RCS were centralized:

    • A single location stored all the files and their history

    • People working with those files checked out a single revision and committed changes to the central repository.

    • RCS, CVS, and SVN are classic examples of centralized systems.

  • In the last few years, there has been a shift to decentralized RCS.

    • Authors and contributors still use a central repository

    • Everybody can have a complete history, work on their own local branches, and merge changes as needed.

    • Bazaar-NG and Git are more recent and decentralized tools.

Managing Configuration Files

Check Out a Git Project
  • Use git clone to check out a project from a Git repository into a working directory on the local system.

    • The Git repository can be hosted locally or on a remote server.

  • The path name is specified as the required argument to the clone command.

  • The optional second argument (config-files in this example), is the directory to create with the Git working directory.

    [student@server1 ~]$ git clone /var/local/config-files.repo config-files
    Initialized empty Git repository in /home/user/config-files/.git/
  • To clone a remote Git repository, the path name would include a user@hostname prefix to specify an SSH account to use to access the remote repository:

    git clone gituser@demo:/var/local/config-files.repo config-files

Managing Configuration Files

Manage Changes Locally with Git
  1. Define who you are with these two global settings:

    [student@server1 ~]$ git config --global user.email student@server1.example.com
    [student@server1 ~]$ git config --global user.name 'Sally Student'

    The above commands create a configuration file called ~/.gitconfig with the appropriate entries using correct syntax.

  2. As you update configuration files, use git status to query Git about the status of your updates.

    Sample query output
    [student@server1 ~]$ cd config-files/
    [student@server1 config-files]$ git status
    # On branch master
    nothing to commit (working directory clean)
    [student@server1 config-files]$ vim hosts
    [student@server1 config-files]$ git status
    # On branch master
    # Changed but not updated:
    #   (use "git add <file>..." to update what will be committed)
    #   (use "git checkout -- <file>..." to discard changes in working directory)
    #
    #   modified:   hosts
    #
    no changes added to commit (use "git add" and/or "git commit -a")
    
    Commit the changes to the repository with git commit -a command
    
    [student@server1 config-files]$ git commit -a
    [master 2af592d] Added an entry for host to hosts file.
     1 files changed, 1 insertions(+), 0 deletions(-)
    [student@server1 config-files]$ git status
    # On branch master
    # Your branch is ahead of 'origin/master' by 1 commit.
    #
    nothing to commit (working directory clean)
    [student@server1 config-files]$ ls
    cups       hosts.allow  issue.net       ssh          yum.conf
    host.conf  hosts.deny   redhat-release  sysctl.conf  yum.repos.d
    hosts      issue        resolv.conf     yum

Managing Configuration Files

Manage Changes Locally with Git
  1. Git does track some file attributes, but it does not preserve file ownership.

    Example: These files were originally owned by root with a group ownership of lp
    [student@server1 config-files]$ ls -l cups/
    total 19
    -rw-rw-r--. 1 student student    0 Sep  8 22:16 classes.conf
    ... Output omitted ...
  2. Use`diff` to display the differences in the file contents:

    [student@server1 config-files]$ git diff
    [student@server1 config-files]$ vim hosts
    [student@server1 config-files]$ git diff
    diff --git a/hosts b/hosts
    index 0a6354e..d008514 100644
    --- a/hosts
    +++ b/hosts
    @@ -1,3 +1,4 @@
     127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
     ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
    -192.168.0.250  demo    demo.example.com
    +192.168.0.250  demo demo.example.com
    +192.168.0.254  instructor instructor.example.com

Managing Configuration Files

Manage Changes Locally with Git
  1. Once you are satisfied with your changes, use git commit -a to store them in the repository.

    • A best practice is to save changes that leave the files in a "working" state.

    • The -a option applies the changes to all of the modified files at or below your current working directory.

      [student@server1 config-files]$ git commit -a -m 'added a hosts entry for instructor'
      [master e0c84d2] added a hosts entry for instructor
       1 files changed, 2 insertions(+), 1 deletions(-)

Managing Configuration Files

Manage Changes Locally with Git
  1. If you accidentally delete a file, use git checkout to recover it.

    • git checkout requires the file to recover as an argument, otherwise it only lists missing files.

    • You can also use git checkout to revert to the repository’s version of a file when working changes need to be discarded.

      [student@server1 config-files]$ ls
      cups       hosts.allow  issue.net       ssh          yum.conf
      host.conf  hosts.deny   redhat-release  sysctl.conf  yum.repos.d
      hosts      issue        resolv.conf     yum
      [student@server1 config-files]$ rm sysctl.conf
      [student@server1 config-files]$ git status
      # On branch master
      # Your branch is ahead of 'origin/master' by 2 commits.
      #
      # Changed but not updated:
      #   (use "git add/rm <file>..." to update what will be committed)
      #   (use "git checkout -- <file>..." to discard changes in working directory)
      #
      #   deleted:    sysctl.conf
      #
      no changes added to commit (use "git add" and/or "git commit -a")
      [student@server1 config-files]$ git checkout
      D   sysctl.conf
      Your branch is ahead of 'origin/master' by 2 commits.
      [student@server1 config-files]$ ls
      cups       hosts.allow  issue.net       ssh       yum.repos.d
      host.conf  hosts.deny   redhat-release  yum
      hosts      issue        resolv.conf     yum.conf
      [student@server1 config-files]$ git checkout -- sysctl.conf
      [student@server1 config-files]$ ls
      cups       hosts.allow  issue.net       ssh          yum.conf
      host.conf  hosts.deny   redhat-release  sysctl.conf  yum.repos.d
      hosts      issue        resolv.conf     yum

Managing Configuration Files

Manage Changes Locally with Git
  1. Use git add to add new files and git rm to remove existing files from a project.

    • Changes are not saved to the repository until you commit them.

      [student@server1 config-files]$ git rm yum.conf
      rm 'yum.conf'
      [student@server1 config-files]$ git status
      # On branch master
      # Your branch is ahead of 'origin/master' by 2 commits.
      #
      # Changes to be committed:
      #   (use "git reset HEAD <file>..." to unstage)
      #
      #   deleted:    yum.conf
      #
      [student@server1 config-files]$ ls
      cups       hosts.allow  issue.net       ssh          yum.repos.d
      host.conf  hosts.deny   redhat-release  sysctl.conf
      hosts      issue        resolv.conf     yum

Managing Configuration Files

Manage Changes Locally with Git
  1. If you issue git rm by mistake, the following commands show how to recover from this:

    [student@server1 config-files]$ git reset HEAD yum.conf
    Unstaged changes after reset:
    M   yum.conf
    [student@server1 config-files]$ git status
    # On branch master
    # Your branch is ahead of 'origin/master' by 2 commits.
    #
    # Changed but not updated:
    #   (use "git add/rm <file>..." to update what will be committed)
    #   (use "git checkout -- <file>..." to discard changes in working directory)
    #
    #   deleted:    yum.conf
    #
    no changes added to commit (use "git add" and/or "git commit -a")
    [student@server1 config-files]$ ls
    cups       hosts.allow  issue.net       ssh          yum.repos.d
    host.conf  hosts.deny   redhat-release  sysctl.conf
    hosts      issue        resolv.conf     yum
    [student@server1 config-files]$ git checkout -- yum.conf
    [student@server1 config-files]$ git status
    # On branch master
    # Your branch is ahead of 'origin/master' by 2 commits.
    #
    nothing to commit (working directory clean)
    [student@server1 config-files]$ ls
    cups       hosts.allow  issue.net       ssh          yum.conf
    host.conf  hosts.deny   redhat-release  sysctl.conf  yum.repos.d
    hosts      issue        resolv.conf     yum

Managing Configuration Files

Manage Changes Locally with Git
  1. Continue the process of removing the yum.conf configuration file:

    [student@server1 config-files]$ git rm yum.conf
    rm 'yum.conf'
    [student@server1 config-files]$ git commit -a -m 'removed yum.conf'
    [master 97ffb60] removed yum.conf
     1 files changed, 0 insertions(+), 23 deletions(-)
     delete mode 100644 yum.conf
  2. Use a similar process to add a file into the repository:

    1. Create the new file.

    2. Use git add.

    3. Use git commit.

      [student@server1 config-files]$ touch newfile
      [student@server1 config-files]$ git status
      # On branch master
      # Your branch is ahead of 'origin/master' by 3 commits.
      #
      # Untracked files:
      #   (use "git add <file>..." to include in what will be committed)
      #
      #   newfile
      nothing added to commit but untracked files present (use "git add" to track)
      [student@server1 config-files]$ git add newfile
      [student@server1 config-files]$ git commit -m 'added newfile'
      [master 3ff06a5] added newfile
       0 files changed, 0 insertions(+), 0 deletions(-)
       create mode 100644 newfile

Managing Configuration Files

Manage Changes in a Central Git Repository
  1. Paths that reference a central repository include a user@hostname prefix to specify the SSH user account and host name used to access the Git repository.

    Example: Authenticating as gituser on the host demo to access a repository in /var/local/config-files.repo

[student@server1 ~]$ git clone gituser@demo:/var/local/config-files.repo config-files
Initialized empty Git repository in /home/user/config-files/.git/
gituser@demo's password:
remote: Counting objects: 32, done.
remote: Compressing objects: 100% (27/27), done.
Receiving objects: 100% (32/32), 8.12 KiB, done.
Resolving deltas: 100% (3/3), done.
remote: Total 32 (delta 3), reused 0 (delta 0)

Managing Configuration Files

Manage Changes in a Central Git Repository
  1. Once the working directory has been created, use Git commands to update the local copy of the repository.

    [student@server1 ~]$ cd config-files/
    [student@server1 config-files]$ vim hosts
    [student@server1 config-files]$ git diff
    diff --git a/hosts b/hosts
    index 849c10d..64809df 100644
    --- a/hosts
    +++ b/hosts
    @@ -1,2 +1,4 @@
     127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
     ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
    +
    +192.168.0.250  demo demo.example.com
    [student@server1 config-files]$ git commit -a -m 'added a hosts entry for demo'
    [master 4fc2335] added a hosts entry for demo
     1 files changed, 2 insertions(+), 0 deletions(-)
    [student@server1 config-files]$ git status
    # On branch master
    # Your branch is ahead of 'origin/master' by 1 commit.
    #
    nothing to commit (working directory clean)
    [student@server1 config-files]$ git rm yum.conf
    rm 'yum.conf'
    [student@server1 config-files]$ git commit -m 'removed yum.conf'
    [master 94f601c] removed yum.conf
     1 files changed, 0 insertions(+), 23 deletions(-)
     delete mode 100644 yum.conf

Managing Configuration Files

Manage Changes in a Central Git Repository
  1. Once you are satisfied with the changes you committed to the working repository, use push to send the changes to the original repository.

    SSH authentication must take place in the absence of SSH user keys.
    [student@server1 config-files]$ git push
    gituser@demo's password:
    Counting objects: 7, done.
    Compressing objects: 100% (5/5), done.
    Writing objects: 100% (5/5), 562 bytes, done.
    Total 5 (delta 3), reused 0 (delta 0)
    To gituser@demo:/var/local/config-files.repo
       fbaa30e..94f601c  master -> master

Managing Configuration Files

Manage Changes in a Central Git Repository
  1. Use pull to retrieve updates made by others.

    If the changes do not conflict or overlap, others' changes are applied to the files in the local repository without human interaction:
    [student@server1 ~]$ cd config-files/
    [student@server1 config-files]$ vim hosts
    [student@server1 config-files]$ git commit -a -m 'add desktop2'
    [master 9c41d10] add desktop2
     1 files changed, 1 insertions(+), 0 deletions(-)
    [student@server1 config-files]$ git push
    gituser@demo's password:
    Counting objects: 5, done.
    Compressing objects: 100% (3/3), done.
    Writing objects: 100% (3/3), 322 bytes, done.
    Total 3 (delta 2), reused 0 (delta 0)
    To gituser@demo:/var/local/config-files.repo
       94f601c..9c41d10  master -> master
    [student@server1 config-files]$ git pull
    gituser@demo's password:
    remote: Counting objects: 10, done.
    remote: Compressing objects: 100% (6/6), done.
    remote: Total 6 (delta 4), reused 0 (delta 0)
    Unpacking objects: 100% (6/6), done.
    From demo:/var/local/config-files.repo
       9c41d10..aca2c25  master     -> origin/master
    Updating 9c41d10..aca2c25
    Fast-forward
     hosts |    1 +
     1 files changed, 1 insertions(+), 0 deletions(-)
    [student@server1 config-files]$ cat hosts
    127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
    ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
    
    192.168.0.250   demo demo.example.com
    192.168.0.1 desktop1 desktop1.example.com
    192.168.0.2 desktop2 desktop2.example.com

Managing Configuration Files

Using Git to Merge Conflicting Changes
  • To minimize conflicts, pull committed changes by others before making changes of your own.

  • Conflicts will still arise if two administrators update overlapping parts of the same file.

    Example: Steps taken and changes made by a local user on resolv.conf.

[student@server1 config-files]$ git pull
gituser@demo's password:
Already up-to-date.
[student@server1 config-files]$ cat resolv.conf
# Generated by NetworkManager
domain example.com
search example.com
nameserver 192.168.0.254
[student@server1 config-files]$ vim resolv.conf
[student@server1 config-files]$ cat resolv.conf
# Generated by NetworkManager
domain example.com
search example.com
nameserver 192.168.0.254
nameserver 127.0.0.1
[student@server1 config-files]$ git commit -a -m 'added localhost entry'
[master 2ae15fe] added localhost entry
 1 files changed, 1 insertions(+), 0 deletions(-)

Managing Configuration Files

Sample error message
[student@server1 config-files]$ git push
gituser@demo's password:
To gituser@demo:/var/local/config-files.repo
 ! [rejected]        master -> master (non-fast-forward)
error: failed to push some refs to 'gituser@demo:/var/local/config-files.repo'
To prevent you from losing history, non-fast-forward updates were rejected
Merge the remote changes before pushing again.  See the 'Note about
fast-forwards' section of 'git push --help' for details.

Managing Configuration Files

Sample conflict resolution
[student@server1 config-files]$ git pull
gituser@demo's password:
remote: Counting objects: 5, done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 3 (delta 1), reused 0 (delta 0)
Unpacking objects: 100% (3/3), done.
From demo:/var/local/config-files.repo
   aca2c25..09ba1fe  master     -> origin/master
Auto-merging resolv.conf
CONFLICT (content): Merge conflict in resolv.conf
Automatic merge failed; fix conflicts and then commit the result.
[student@server1 config-files]$ cat resolv.conf
search example.com
nameserver 192.168.0.254
<<<<<<< HEAD
nameserver 127.0.0.1
=======
nameserver 192.168.0.253
>>>>>>> 09ba1fecf9d47c3e43b0c21f7f3b3098b596cb34
[student@server1 config-files]$ vim resolv.conf
[student@server1 config-files]$ git commit -a -m 'added localhost entry with .253 entry'
[master 4f895ad] added localhost entry with .253 entry
[student@server1 config-files]$ git push
gituser@demo's password:
Counting objects: 10, done.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (6/6), 637 bytes, done.
Total 6 (delta 3), reused 0 (delta 0)
To gituser@demo:/var/local/config-files.repo
   09ba1fe..4f895ad  master -> master

Managing Configuration Files

Creating a Local Git Repository
  1. Create a directory to serve as the repository, change into that directory, and then use git init to initialize the repository and create a .git subdirectory.

    [student@server1 ~]$ mkdir my-files
    [student@server1 ~]$ cd my-files
    [student@server1 my-files]$ git init
    Initialized empty Git repository in /home/user/my-files/.git/
  2. Use git add in combination with git commit to populate the repository with files.

    [student@server1 my-files]$ cp ~/.bashrc bashrc
    [student@server1 my-files]$ cp ~/.bash_profile bash_profile
    [student@server1 my-files]$ git add .
    [student@server1 my-files]$ git commit -m 'account config files'
    [master (root-commit) 1cb15e8] account config files
     2 files changed, 20 insertions(+), 0 deletions(-)
     create mode 100644 bash_profile
     create mode 100644 bashrc
    [student@server1 my-files]$ vim bash_profile
    [student@server1 my-files]$ git diff
    diff --git a/bash_profile b/bash_profile
    index 3dc099a..a3679c5 100644
    --- a/bash_profile
    +++ b/bash_profile
    @@ -8,5 +8,6 @@ fi
     # User specific environment and startup programs
    
     PATH=$PATH:$HOME/bin
    +EDITOR=vim
    
    -export PATH
    +export PATH EDITOR
    [student@server1 my-files]$ git commit -a -m 'defined EDITOR environment variable'
    [master 41a0605] defined EDITOR environment variable
     1 files changed, 2 insertions(+), 1 deletions(-)
References
  • git(1) and gittutorial(7) man pages

  • /usr/share/doc/git-*/

  • General Git Documentation

Change Management Best Practices

Provide early and consistent communication
  • Notify everyone as early as possible to provide time to prepare and adjust.

    • Extra time allows for everyone involved to plan and collaborate for a smooth change.

  • Establish a change management/control meeting, process, or board that includes representatives of all the regular stakeholders in an organization.

    • Changes should be discussed and signed off by all parties

    • Provide a policy for minor recurring changes

  • Provide a prioritization policy for changes.

Timing is everything
  • Make sure that all parties involved agree with the time frame for the change.

    • Deploying when people are out of the office or during a very busy part of a sales cycle could lead to disaster.

  • Provide a streamlined process for emergency changes.

Change Management Best Practices

Provide a solid execution strategy
  • Detail each step that will be required to make the change.

    • Document expected responses or return codes

    • Document errors which may be false positives

Make the change desirable and communicate
  • Help all involved to understand the need for the change by discussing the current issues and the intended outcome of the change.

  • Be ready to provide and present the facts regarding the change.

  • Break down the benefits of the change to show each stakeholder how they will benefit.

  • Making the change more relevant on a personal level will help to encourage change.

  • Explain how this change will help the overall organization achieve its goals.

Change Management Best Practices

Prepare to back out
  • Should something go wrong, have a detailed back-out plan that will bring you to the exact state you were in before the change.

    • Back up configuration files and data before making changes

    • Provide a plan and be prepared to test whether the back-out was successful

  • Test back-out procedures in a development/test environment.

Iron out the kinks ahead of time
  • Do not deploy any changes that are untested.

  • Maintain a development and/or test environment that is identical to production.

  • If possible, changes should be tested by a team that is independent from the party requesting the change.

  • Do not assume that because a vendor provides a patch, fix, or recommendation it is foolproof.

    • Trust but verify

Change Management Best Practices

Monitor the change as it is being implemented
  • When possible, make changes in a phased approach.

    • Closely watch all phases and only continue to the next phase when absolutely sure the current phase is complete.

Provide a plan for notification and testing after changes are complete
  • Make sure all stakeholders have solid regression testing plans ready for execution.

    • Regression testing requires testing all existing and new functionality.

  • Be prepared to notify stakeholders when changes are complete and the environment being changed is ready for testing.

Implement a review process after the change
  • This helps bring to light any “lessons learned” and issues with the process or organization implementing the process.

  • This will also identify any issues relating to the change which may still require attention.

Configuration Management

Summary
  • Configuration management is the task of tracking and controlling changes to source, scripts, or configuration files.

  • Configuration management practices can include revision control and the establishment of baselines.

    • If something goes wrong, it can determine what was changed and who changed it.

    • If a configuration is working well, it can determine how to replicate changes across many hosts.

  • When testing changes in a test or development environment, you can check them into a revision control system such as Git.

    • Enables you to track changes, who made changes, and why changes were made.

  • Environments that do not adhere to revision control are likely to experience more problems than environments that adhere to revision control policies.

Module Completion

Nice job!

Click the button below to complete this module of the course: